Decouple repository root resolver from internal logic #2957

teresaromero · 2025-09-25T14:16:22Z

Resolves #2797

Summary

Refactored the linked files system to use explicit os.Root parameters for secure repository access. This change fixes a bug where linked files were copied to incorrect directories and reveals a secondary issue where external fields were resolved before linked files were included in the build bundle.

Problems Fixed

Linked files copied to wrong directory: The workDir in LinksFS was treated inconsistently (absolute vs relative), causing incorrect path resolution. Now standardized to always be relative to repository root.
External fields resolved too early: The build process resolved external fields before copying linked files, causing missing field definitions. Reordered build steps to copy linked files first.

Key Changes

Repository Root Handling:

CreateLinksFSFromPath() now requires explicit *os.Root parameter
All callers must find and pass repository root explicitly
Added proper cleanup with defer repositoryRoot.Close()

API Changes:

FindPackageRoot() signature: (string, bool, error) → (string, error)
Added ErrPackageRootNotFound sentinel error
BuildOptions.PackageRoot renamed to PackageRootPath and added RepositoryRoot field
Package installer Options.RootPath renamed to PackageRootPath and added RepositoryRoot field

Build Process:

License file resolution now uses repository root securely
Linked files included before resolving external fields
ELASTIC_PACKAGE_REPOSITORY_LICENSE now expects paths relative to repository root

Breaking Changes

Functions that previously auto-discovered repository root now require it as a parameter:

CreateLinksFSFromPath(repositoryRoot, workDir)
BuildPackage() requires RepositoryRoot in options
NewForPackage() requires RepositoryRoot in options

Testing

All existing tests updated and passing
New test coverage for license file handling and edge cases
Manual verification confirms fixes for both bugs

…d clarity

…processing

…p files

… and tests

internal/elasticsearch/ingest/datastream.go

… and update options in rally and stream runners

…nk file paths

internal/packages/archetype/data_stream_test.go

internal/docs/links_map.go

internal/files/linkedfiles.go

internal/docs/links_map_test.go

internal/builder/packages_test.go

internal/files/linkedfiles.go

internal/builder/packages.go

…factor tests and added readme note

README.md

internal/builder/packages.go

teresaromero · 2025-10-09T13:51:07Z

@mrodm i've updated the code with your feedback. i usually mark resolved the comments that have been addressed, please re-open if you find anything pending.

thanks

jsoriano

It looks great, it seems to solve links resolution and does a good refactor to allow injection of the path to the root repository.

I did some tests and found some issues, commented in the files, though maybe some of them can be solved in follow ups.

Many comments are nitpicking, no need to do anything about them if you prefer current code.

internal/benchrunner/runners/rally/options.go

cmd/benchmark.go

internal/builder/packages.go

internal/resources/fleetpolicy.go

internal/resources/fleetpolicy_test.go

internal/testrunner/runners/asset/runner.go

internal/files/linkedfiles.go

internal/builder/packages.go

…ling

… files

…seTextFile

…and installer

…use value receiver instead of pointer

… handling in tests

…th to PackageRootPath

…and update related tests

…pdate related tests

…esolve external fields

elasticmachine · 2025-10-22T07:50:40Z

💚 Build Succeeded

Buildkite Build
Commit: 94a16d7

History

💔 Build #6396 failed 7570581
💔 Build #6390 failed 24a9f73
💔 Build #6354 failed d08ed58
💔 Build #6353 failed 8af2c99
💚 Build #6346 succeeded 3b32fcc

jsoriano

Great changes here 👍

jsoriano · 2025-10-22T07:45:31Z

internal/elasticsearch/ingest/datastream.go

 // elasticsearch/ingest_pipeline under the provided data stream path. The names
 // of the pipelines are decorated with the provided nonce.
-func LoadIngestPipelineFiles(dataStreamPath string, nonce int64) ([]Pipeline, error) {
+func LoadIngestPipelineFiles(dataStreamPath string, nonce int64, repositoryRoot *os.Root) ([]Pipeline, error) {


I wonder if these methods should just receive an FS, instead of receiving the root and creating its own 🤔 For another refactor in any case.

Yes, i think there is room for improvement in following refactor. Also i've notice some of the code around the builder, package paths, build paths, etc... that are a bit confusing so perhaps when we work on the feature to allow builds in any target directory we could have some time to improve that part.

teresaromero added 7 commits September 25, 2025 15:51

improve comments for Link struct and its fields in linkedfiles.go

75dae97

refactor: enhance newLinkedFile function for better error handling an…

83e13ad

…d clarity

test: add unit test for newLinkedFile function to validate link file …

9139bb2

…processing

decoupled repoRoot from lint cmd and docs

60672b2

update repository root handling across multiple components

211c7c9

standardize repository root handling across build, lint, and links ma…

0b61910

…p files

ensure repository root is properly closed in multiple command actions…

ce07ec3

… and tests

teresaromero commented Sep 25, 2025

View reviewed changes

internal/elasticsearch/ingest/datastream.go Outdated Show resolved Hide resolved

teresaromero added 10 commits September 29, 2025 09:41

add repoRoot parameter to build command action

0deae99

update linked file handling to use absolute paths and improve clarity

1fc643a

add copyright notice to packages_test.go file

6820b49

refactor linkedfiles_test.go to use os.MkdirAll for directory creation

2c16cab

implement repository root handling in benchmark and install commands,…

403527e

… and update options in rally and stream runners

read repoRoot at test runner cmd

96d5129

refactor linkedfiles_test.go to use filepath.Join for constructing li…

90ba403

…nk file paths

inject repoRoot to needed test options

aeeee55

Update data stream tests to include repoRoot parameter

385fc89

Ensure repoRoot is closed in all data stream test cases

9edca9b

mrodm reviewed Sep 30, 2025

View reviewed changes

internal/packages/archetype/data_stream_test.go Outdated Show resolved Hide resolved

Remove unnecessary directory change in createAndCheckDataStream function

c60b836

teresaromero marked this pull request as ready for review September 30, 2025 13:01

mrodm reviewed Oct 3, 2025

View reviewed changes

teresaromero added 2 commits October 6, 2025 09:03

Merge branch 'main' into 2797-detach-repo-root

679c566

refactor at copyLicenceTextFile to ensure license inside reporoot, re…

ef3497f

…factor tests and added readme note

mrodm reviewed Oct 7, 2025

View reviewed changes

README.md Show resolved Hide resolved

internal/builder/packages.go Show resolved Hide resolved

internal/builder/packages.go Outdated Show resolved Hide resolved

internal/builder/packages.go Outdated Show resolved Hide resolved

teresaromero added 5 commits October 8, 2025 14:09

Merge branch 'main' into 2797-detach-repo-root

1f591bf

refactor linkefiles management to use relative paths and root

ec2a7e1

revert linksDefinitionsFilePath from public to private

169777f

add validation for RepoRoot in Options

9229664

clarify variable name for included file path in newLinkedFile function

0436a37

teresaromero added 4 commits October 9, 2025 11:38

refactor findRepositoryLicensePath to use repoRoot

5d6ef55

refactor copyLicenseTextFile to use repoRoot.Stat

ecfeb9b

refactor FindPackageRoot and handle error on FindPackageRootFrom calls

4412b0b

remove repoRoot.Write and Read

9b7a721

jsoriano reviewed Oct 14, 2025

View reviewed changes

teresaromero added 21 commits October 15, 2025 10:24

Merge branch 'main' into 2797-detach-repo-root

091d117

rename repoRoot to repositoryRoot for clarity

66274da

fix: correct grammar in comments for license file handling

5429121

update copyLicenseTextFile function to improve clarity and error hand…

b3e73ac

…ling

enhance findRepositoryLicensePath function to check for empty license…

563522f

… files

improve path handling in TestFindRepositoryLicense and TestCopyLicen…

bfe509e

…seTextFile

rename test function and update comment for clarity

59436db

remove unnecessary blank lines in test files for improved readability

248a704

rename PackageRoot to PackageRootPath for consistency across builder …

c74c6c8

…and installer

add repository root handling in benchmark and test runner commands

68ed0d8

rename newLinkMap to newEmptyLinkMap and revert related functions to …

54d8502

…use value receiver instead of pointer

update findRepositoryLicensePath to use os.ReadFile and improve error…

3b32fcc

… handling in tests

remove RepositoryRoot references from FleetAgentPolicy, rename RootPa…

8af2c99

…th to PackageRootPath

refactor copyLicenseTextFile to enforce absolute target license path …

d08ed58

…and update related tests

refactor build package handling to inject buildDir

c4a5205

refactor readme generation to include sourceFilesRoot parameter and u…

5d84f01

…pdate related tests

Merge branch 'main' into 2797-detach-repo-root

c8b06ad

streamline linked files inclusion in BuildPackage function prior to r…

102dfcd

…esolve external fields

Merge branch 'main' into 2797-detach-repo-root

24a9f73

restore zipped built package path to build/packages

7570581

Add repository root handling to test runner and installer

94a16d7

teresaromero mentioned this pull request Oct 22, 2025

Upgrade go version to 1.25.3 for root.ReadFile and root.WriteFile #3015

Open

jsoriano approved these changes Oct 22, 2025

View reviewed changes

Decouple repository root resolver from internal logic #2957

Are you sure you want to change the base?

Decouple repository root resolver from internal logic #2957

Uh oh!

Conversation

teresaromero commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problems Fixed

Key Changes

Breaking Changes

Testing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

teresaromero commented Oct 9, 2025

Uh oh!

jsoriano left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Oct 22, 2025

💚 Build Succeeded

History

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

jsoriano Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

teresaromero Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

teresaromero commented Sep 25, 2025 •

edited

Loading

jsoriano left a comment •

edited

Loading